Skip to content

feat(routing): add opt-in outcome-bound prompt cache affinity - #40

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-3from
feat/prompt-cache-affinity
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-3from
feat/prompt-cache-affinity

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds outcome-bound prompt cache affinity behind PROMPT_CACHE_AFFINITY_ENABLED, which defaults to false. While it is off, request and response bytes, headers, credential order and retry behaviour are unchanged.

  • With the flag on, authenticated managed automatic text routes prefer the credential that actually completed a reusable prefix:
    • managed Chat Completions;
    • automatic aliases translated through Responses or Messages.
  • A binding is created only from a classified, complete, successful attempt:
    • Stream headers and early deltas never bind.
    • Failed, incomplete, interrupted and cancelled attempts evict only the matching binding.
    • Affinity adds no retry permission.
  • Preference only reorders keys the pool already allows. Model cooldown, credential rest, grants, budgets, context capability and fallback eligibility stay authoritative, and ordered automatic routes keep their configured provider order.
  • Storage holds hashes only and is bounded: TTL PROMPT_CACHE_AFFINITY_TTL_SECONDS (default 900, from 1 to 86,400), with an LRU limit. Bindings are isolated by principal, session, role lane, route, opening prefix, tool schema and policy revision.
  • Cost evidence uses measured cache-read savings and labels rebuild estimates as estimates. It never changes policy automatically.
  • Explicit models, raw proxy paths and native roleplay are unchanged.

Details are in docs/prompt-cache-affinity.md.

Integration

The Wave 4 integration PR connects two remaining hooks: AffinityScope.order_candidates for a tier-aware scheduler, and the intelligence transport's completion callback. Until then, affinity applies to the managed unified route and its automatic aliases only. routes/unified.py is 999 lines; the edits are small call sites.

Shared registrations

  • Added PROMPT_CACHE_AFFINITY_ENABLED and PROMPT_CACHE_AFFINITY_TTL_SECONDS to the Container env keys.

Tests

  • tests/test_prompt_cache_affinity.py: new. It covers:
    • disabled and invalid settings leaving everything unchanged;
    • binding only after classified completion;
    • stream and abort cases;
    • eviction scoped to the failed candidate;
    • cooldown precedence;
    • principal, session and revision isolation;
    • bounded hash-only storage;
    • the registered managed alias preserving bytes.
  • Full Python suite: 3,594 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,349 passed.
  • cf:migrations:check, test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.
  • GitHub's python and node jobs currently fail at the dependency-audit step on every branch, including main, because of new advisories. The tests themselves don't run there. A separate dependency PR fixes this.

This is part of Wave 4. It is based on integrate/gateway-waves-1-3, and the Wave 4 integration PR adds the README line.

Prefer the credential that actually completed a reusable prefix on authenticated managed automatic routes behind PROMPT_CACHE_AFFINITY_ENABLED (default off). Bindings come only from classified successful attempts, are hash-only and bounded by TTL and LRU, and never override cooldown, grants, budgets or fallback eligibility.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 10a4cfe6-c50c-43e4-8079-e6417b15b5a9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 22d205c Oct 09 2026, 12:23 AM

@ALikesToCode
ALikesToCode merged commit 07e45d9 into integrate/gateway-waves-1-3 Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/prompt-cache-affinity branch October 26, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant